Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

extensions framework + UEFI x86 + rpi4b + core changes/fixes + doc generator #3282

Merged

Conversation

rpardini
Copy link
Member

@rpardini rpardini commented Nov 22, 2021

  • "if at first you don't succeed, hide all evidence that you tried", but done wrong.
  • v2 of all patches forced pushed, fixing stuff and adressing reviews below.
  • v3 force pushed. rpi4b and uefi fixes. thanks for all the screenshots and tests!
  • v4 force pushed. adding uefi-arm64 and uefi-x86 mainline kernel build (edge builds)
  • v5 is here. many fixes for kernels, firmwares, etc. mainline kernel for rpi4b (edge builds)
    • avoid conflict between armbian-firmware and linux-firmware-raspi2 by removing the later.
    • use separate KERNELDIR for these 3 new kernels, until we're sure they're safe to mix.
  • v6: many fixes. hopefully final version, pending test images for edge and current
  • v7: small fixes. final version. I'd say ready for merge.

extensions framework (née "fragments")

  • this should actually change nothing at this point, just add capabilities
  • the framework is implemented in lib/extensions.sh
  • the "if function x exists then call x" replaced with call_extension_method()
    • +inline documentation
    • +compatibility names

extensions framework; meta-extensions: auto-docs and sample extension gen

  • 2 extensions dealing with extensibility itself
    • detect-unused-extensions: shows which extensions are enabled, but never called.
    • gen-sample-extension-docs: generates a sample empty extension & Markdown documentation for extensions

new extension methods and features via config variables in core Armbian

  • SKIP_EXTERNAL_TOOLCHAINS=yes - does not download or use any linaro toolchains, only build host-installed ones
  • SKIP_BOOTSPLASH=yes - does not patch kernel for splash file
  • EXTRA_BSP_NAME=xyz - allows for BSP variants, useful for when extensions modify the BSP
  • EXTRA_ROOTFS_MIB_SIZE=x - add x mib's to rootfs size, for use with very small images
  • KERNEL_EXTRA_TARGETS - what extra targets to make kernel for, default to "modules dtbs"
  • BOOTCONFIG=none - does not build nor install u-boot; also doesn't handle bootscripts et al
  • unset KERNELSOURCE - does not build nor install kernel, nor build initrd, nor build nor install firmware
  • ARMHF_ARCH=skip - does not add armhf to apt/dpkg, thus pure arm64
  • SKIP_ARMBIAN_REPO=yes - results in armbian.list.disabled in the final image
  • define APT_EXTRA_DIST_PARAMS with apt-cacher-ng options and use it for PACKAGE_LIST_INSTALL/REMOVE et al
  • initial support for targeting x86/amd64 UEFI and BIOS
    • some do's/don'ts for x86/amd64, like a different UBUNTU_MIRROR default
    • GPT/EFI(ESP) partitions (fat, UEFISIZE=256 to enable, mount UEFI_MOUNT_POINT=/boot/efi, first on disk but ends
      up at $uefipart=15)
    • GPT/BIOS partitions (fat, BIOSSIZE=1 to enable, second on disk but ends up at partition 14)
    • UEFI_FS_LABEL="armbiefi" - to set the FAT label for the EFI partition, visible in Win/Mac
    • hard-requires gdisk package host-side
  • add add_host_dependencies() extension method; fill EXTRA_BUILD_DEPS="pkg pkg2" to install to host before toolchains
    download
  • add pre_prepare_partitions() extension method, for custom partition size calculations
  • add create_partition_table() extension method, used to do full-custom partitioning if USE_HOOK_FOR_PARTITION=yes
  • add post_create_partitions() extension method, mostly for easy debugging
  • add post_write_sdcard() extension method, where you can also set SKIP_VERIFY=yes to skip sdcard verification
  • add pre_install_kernel_debs() and post_install_kernel_debs() extension methods.
  • multiple fixes to bsp to avoid spurious errors when files are not where it expects
  • v4: detect update-initramfs failure and abort build with useful message if it does
  • v4: show useful stacktrace in exit_with_error() and unmount_on_exit()
    • if ERROR_DEBUG_SHELL=yes, drop into a shell before unmounting/deleting everything, so we can inspect what went
      wrong
  • v4: display a message before apt-get remove PACKAGE_LIST_BOARD_REMOVE packages, so any errors while removing are
    easy to understand
  • v4: preserve kernel .config's dates when copying

extensions framework; refactor tool fetching/building into extensions

  • a few examples of core refactoring using extensions
  • sunxi-tools extension, enabled by 2 different sunxi family includes ("reuse" example)
  • marvel-tools extension, enabled by 2 different mvebu family includes
  • rkbin-tools extension, enabled by rockship64_common family include
  • amlogic-fip/c2-blobs stuff refactored directly into meson64_common.inc ("single-use" example)
  • removed the 'testings' fetch_from_repo completely since not used anywhere.

.wip's for UEFI arm64 and UEFI/BIOS x86 via new GRUB extension

  • v3: added growroot-awareness to armbian-resize-filesystem
    • the partition-growing part of armbian-resize-filesystem does not deal correctly with the UEFI layout
    • growroot is installed on UEFI images by default, that handles growing partition during initramfs
    • now armbian-resize-filesystem handles resize2fs only, and works.
  • v4: reworked UEFI board/family/include structure:
    • use Distro's linux-generic kernel only for current
    • edge now builds it's own pure-mainline 5.15.y kernel, for both x86 and arm64
      • .config taken from Ubuntu, probably needs tuning for EXTRAWIFI=yes et al
  • v4: introduce SKIP_KERNEL_SYMLINK=yes, tested in builddeb
    • to avoid symlinking kernel; u-boot likes it, but grub and flash-kernel hates it
  • v5: many fixes
  • v7: more small fixes.

.wip for the RaspberryPi 4B via new flash-kernel extension

  • this does not build it's own kernel "yet", but uses default linux-raspi kernel from Ubuntu
  • flash-kernel is not really a bootloader
    • it just prepares kernel et al a FAT partition for booting by the RPi4b bootloader
    • flash-kernel is standard Debian package, but has only been tested on Ubuntu releases
  • it is really only known-working since Hirsute release.
    • Debian's rpi kernel is armhf only, so out of scope here, at least until we add source-built kernels.
  • v3: fixed focal rootfs build. untested.
  • v3: better variable names, preparing for source-built kernel.
  • v5: new edge build with mainline kernel.
  • v6: many fixes and some hacks for packaging and layout, also firmware (using Ubuntu's)

@rpardini
Copy link
Member Author

rpardini commented Nov 22, 2021

This is just a preview. If I have blood left in me, I'll still

  • refactor u-boot itself into an extension, to avoid the BOOTCONFIG=none hack
  • refactor sunxi-tools and rkbin-tools and amlogic-fip/c2blobs stuff into their own extensions, as to only download/build when they're actually needed Done!
  • refactor specific toolchain requirements into their own families, using built-in extension, avoiding download when unneeded

Copy link
Member

@igorpecovnik igorpecovnik left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I made one quick review and will need to do a few more to understand it well. Generally looks ok.

config/boards/rpi4b.wip Outdated Show resolved Hide resolved
config/boards/rpi4b.wip Outdated Show resolved Hide resolved
config/sources/amd64.conf Outdated Show resolved Hide resolved
@rpardini rpardini force-pushed the extensions-meta-core-uefi-rpi4b branch 3 times, most recently from 6096817 to 417b025 Compare November 22, 2021 21:35
@rpardini
Copy link
Member Author

rpardini commented Nov 22, 2021

Test images are up!
https://github.com/rpardini/armbian-release/releases/tag/20211122g (many more builds here)

@lanefu
Copy link
Member

lanefu commented Nov 23, 2021

alright trying to break things now

@150balbes
Copy link
Contributor

150balbes commented Nov 23, 2021

A quick glance. Questions.

  1. ARMbian is a build system for ARM, why drag x86 here?
  2. I suggest framing all places where new code is added (except for completely new configuration files) with the

/# use new code
if [ USE_EXT_NEW_CODE=yes ] ; then
/# Source the extensions manager library at this point, before sourcing the config.
/# This allows early calls to enable_extension(), but initialization proper is done later.
/# shellcheck source=lib/extensions.sh
source "${SRC}"/lib/extensions.sh
fi

If the existing code is being modified, the old code is placed after "else"

construction and adding the USE_EXT_NEW_CODE variable to the config (any variable name, at your discretion, this is just an example). This will allow at any time (for example, there was an incomprehensible problem with the assembly behavior) to disable all new fragments with one variable in order to have a classic build system and exclude any influence of the new code.

Copy link
Contributor

@150balbes 150balbes left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

information in the discussion thread

@lanefu
Copy link
Member

lanefu commented Nov 23, 2021

Confirmation for Honeycomb and Pyhtium

IMG_20211123_211815
Screenshot_20211123-152027

@rpardini
Copy link
Member Author

First, thanks for the review @150balbes

  1. ARMbian is a build system for ARM, why drag x86 here?

Indeed. x86 was an accident. The main body of code is UEFI support via GRUB for aarch64 (I developed and tested on Ampere VMs, with OVMF/Tianocore firmware), but soon realised that x86 was very easy to add and costed nothing. Right now x86 has a lot of interest, from people with laptops to servers to x86 SBCs. Maybe I should rename the x86 images "AMDbian" just for laughs. PS: See screenshots above! aarch64 UEFI stuff.

if [ USE_EXT_NEW_CODE=yes ] ; then

I understand your concern, but IMHO using these kind of constructs would negate the long-term benefits of having an extensibility framework. With time (and bugfixes, of course) we will trust the extensibility system. For example, try building this branch with BOARD=uefi-arm64 DEBUG_EXTENSION_CALLS=yes LOG_ENABLE_EXTENSION=yes ENABLE_ENTENSIONS=detect-unused-extensions,gen-sample-extension-docs - notice how you can tell when each extensions is called, without having to inspect the code.

I also plan to add strict shellcheck and shfmt PR tests, and run extension code (and only extension code) with set -e by default, to enforce error handling. This way we make the codebase better by refactoring core stuff into extensions, bit by bit.

@NicoD-SBC
Copy link

neofetch RPi4

@rpardini rpardini force-pushed the extensions-meta-core-uefi-rpi4b branch from 417b025 to e5a9762 Compare November 24, 2021 17:39
@rpardini
Copy link
Member Author

v3 force pushed.

  • fixes for rpi4b focal, make clear that Debian won't work, better handle errors.
  • uefi: make armbian-resize-filesystem growroot aware. should fix resize root problem.

@rpardini rpardini force-pushed the extensions-meta-core-uefi-rpi4b branch from e5a9762 to 96a06ae Compare November 27, 2021 00:00
@rpardini
Copy link
Member Author

v4 force pushed.

  • many fixes
  • pure mainline kernels for aarch64 & x86 UEFI

@rpardini rpardini force-pushed the extensions-meta-core-uefi-rpi4b branch from 96a06ae to 054348a Compare November 27, 2021 20:17
@rpardini
Copy link
Member Author

v6: with many fixes. things should work with both current and edge.

- this should actually change nothing at this point, just add capabilities
- the framework is implemented in lib/extensions.sh
- the "if function x exists then call x" replaced with call_extension_method()
  - +inline documentation
  - +compatibility names

Signed-off-by: Ricardo Pardini <ricardo@pardini.net>
… gen

- 2 extensions dealing with extensibility itself
  - detect-unused-extensions: shows which extensions are enabled, but never called.
  - gen-sample-extension-docs: generates a sample empty extension & Markdown documentation for extensions

Signed-off-by: Ricardo Pardini <ricardo@pardini.net>
- `SKIP_EXTERNAL_TOOLCHAINS=yes` - does not download or use any linaro toolchains, only build host-installed ones
- `SKIP_BOOTSPLASH=yes` - does not patch kernel for splash file
- `EXTRA_BSP_NAME=xyz` - allows for BSP variants, useful for when extensions modify the BSP
- `EXTRA_ROOTFS_MIB_SIZE=x` - add x mib's to rootfs size, for use with very small images
- `KERNEL_EXTRA_TARGETS` - what extra targets to make kernel for, default to "modules dtbs"
- `BOOTCONFIG=none` - does not build nor install u-boot; also doesn't handle bootscripts et al
- `unset KERNELSOURCE` - does not build nor install kernel, nor build initrd, nor build nor install firmware
- `ARMHF_ARCH=skip` - does not add armhf to apt/dpkg, thus pure arm64
- `SKIP_ARMBIAN_REPO=yes` - results in armbian.list.disabled in the final image
- define `APT_EXTRA_DIST_PARAMS` with apt-cacher-ng options and use it for `PACKAGE_LIST_INSTALL/REMOVE` et al
- initial support for targeting x86/amd64 UEFI and BIOS
    - some do's/don'ts for x86/amd64, like a different `UBUNTU_MIRROR` default
    - GPT/EFI(ESP) partitions (fat, `UEFISIZE=256` to enable, mount `UEFI_MOUNT_POINT=/boot/efi`, first on disk but ends
      up at `$uefipart`=15)
    - GPT/BIOS partitions (fat, `BIOSSIZE=1` to enable, second on disk but ends up at partition 14)
    - `UEFI_FS_LABEL="armbiefi"` - to set the FAT label for the EFI partition, visible in Win/Mac
    - hard-requires gdisk package host-side
- add add_host_dependencies() extension method; fill `EXTRA_BUILD_DEPS="pkg pkg2"` to install to host before toolchains
  download
- add pre_prepare_partitions() extension method, for custom partition size calculations
- add create_partition_table() extension method, used to do full-custom partitioning if `USE_HOOK_FOR_PARTITION=yes`
- add post_create_partitions() extension method, mostly for easy debugging
- add post_write_sdcard() extension method, where you can also set `SKIP_VERIFY=yes` to skip sdcard verification
- add post_install_kernel_debs() extension method.
- multiple fixes to bsp to avoid spurious errors when files are not where it expects
- v4: detect `update-initramfs` failure and abort build with useful message if it does
- v4: show useful stacktrace in `exit_with_error`
  - if `ERROR_DEBUG_SHELL=yes`, drop into a shell before unmounting/deleting everything, so we can inspect what went wrong
- v4: display a message before `apt-get remove PACKAGE_LIST_BOARD_REMOVE` packages, so any errors while removing are easy to understand
- v4: preserve kernel .config's dates when copying

Signed-off-by: Ricardo Pardini <ricardo@pardini.net>
- a few examples of core refactoring using extensions
- sunxi-tools extension, enabled by 2 different sunxi family includes ("reuse" example)
- marvel-tools extension, enabled by 2 different mvebu family includes
- rkbin-tools extension, enabled by rockship64_common family include
- amlogic-fip/c2-blobs stuff refactored directly into meson64_common.inc ("single-use" example)
- removed the 'testings' fetch_from_repo completely since not used anywhere.

Signed-off-by: Ricardo Pardini <ricardo@pardini.net>
@rpardini rpardini force-pushed the extensions-meta-core-uefi-rpi4b branch from 054348a to 9afbd5c Compare November 27, 2021 23:05
@rpardini
Copy link
Member Author

I rebased and pushed v6 again (no other changes)

- v3: added `growroot`-awareness to `armbian-resize-filesystem`
    - the partition-growing part of `armbian-resize-filesystem` does not deal correctly with the UEFI layout
    - `growroot` is installed on UEFI images by default, that handles growing partition during initramfs
    - now `armbian-resize-filesystem` handles `resize2fs` only, and works.
- v4: reworked UEFI board/family/include structure:
    - use Distro's `linux-generic` kernel only for `current`
    - `edge` now builds it's own pure-mainline `5.15.y` kernel, for both x86 and arm64
        - `.config` taken from Ubuntu, probably needs tuning for EXTRAWIFI=yes et al
- v4: introduce `SKIP_KERNEL_SYMLINK=yes`, tested in `builddeb`
    - to avoid symlinking kernel; u-boot likes it, but grub and flash-kernel hates it
- v5: many fixes
- v7: more small fixes.

Signed-off-by: Ricardo Pardini <ricardo@pardini.net>
- this does not build it's own kernel "yet", but uses default linux-raspi kernel from Ubuntu
- flash-kernel is not really a bootloader
    - it just prepares kernel et al a FAT partition for booting by the RPi4b bootloader
    - flash-kernel is standard Debian package, but has only been tested on Ubuntu releases
- it is really only known-working since Hirsute release.
  - Debian's rpi kernel is armhf only, so out of scope here, at least until we add source-built kernels.
- v3: fixed focal rootfs build. untested.
- v3: better variable names, preparing for source-built kernel.
- v5: new edge build with pure mainline kernel.
- v6: many fixes and some hacks for packaging and layout, also firmware (using Ubuntu's)

Signed-off-by: Ricardo Pardini <ricardo@pardini.net>
@rpardini rpardini force-pushed the extensions-meta-core-uefi-rpi4b branch from 9afbd5c to d9e5d84 Compare November 28, 2021 13:55
@rpardini
Copy link
Member Author

v7 with "final" small fixes pushed.

@rpardini
Copy link
Member Author

v7 based edge (mainline kernel) test images are up! Using 5.15.5 kernel.

(many more builds here, also current with Distro's kernel and clouds:) 20211128c

@igorpecovnik igorpecovnik changed the base branch from master to AR-1009 November 28, 2021 20:37
@igorpecovnik igorpecovnik merged commit 67b9ab4 into armbian:AR-1009 Nov 28, 2021
@The-going
Copy link
Contributor

The continuation of the conversation in #3298

@rpardini
Copy link
Member Author

rpardini commented Nov 29, 2021

Attention: this PR was NOT merged into master, but into a different/shared branch (for more reviews).
Edit: That branch has now been force-pushed, and #3300 is where the discussion will continue.
So none of the new boards are available to build in master, only in AR-1009 branch. Feedback into #3300 please

@rpardini
Copy link
Member Author

The continuation of the conversation in #3298

That was also closed. New PR #3300 has a review that is equivalent.

@rpardini rpardini deleted the extensions-meta-core-uefi-rpi4b branch December 14, 2021 19:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants